{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yargy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yargy — библиотека для извлечения структурированной информации из текстов на русском языке. Правила описываются контекстно-свободными грамматиками и словарями ключевых слов. Банк готовых правил для имён, дат, адресов и других сущностей доступен в репозитории <a href=\"https://github.com/natasha/natasha\">Natasha</a>.\n",
    "\n",
    "Парсер реализует алгоритм <a href=\"https://en.wikipedia.org/wiki/Earley_parser\">Earley parser</a>. Библиотека написана на чистом Python, поддерживает Python 3.5+ и Pypy 3, использует <a href=\"https://pymorphy2.readthedocs.io/en/latest/\">Pymorphy2</a> для работы с морфологией."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Томита-парсер"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://tech.yandex.ru/tomita/\">Томита-парсер</a> — популярный инструмент для извлечения структурированный информации из текстов на русском языке. Грубо говоря, Yargy — версия Томита-парсера для Python, написанная с нуля:\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<th>Томита-парсер</th>\n",
    "<th>Yargy</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>Разрабатывался много лет внутри Яндекса</td>\n",
    "<td>Open source, разрабатывается сообществом</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>10 000+ строк кода на C++</td>\n",
    "<td>1000+ на Python</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>CLI</td>\n",
    "<td>Python-библиотека</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>Protobuf + конфигурационные файлы</td>\n",
    "<td>Python DSL</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>Нет готовых правил</td>\n",
    "<td><a href=\"https://github.com/natasha/natasha\">Natasha</a> — готовые правила для извлечения имён, дат, адресов и других сущностей</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>Медленный</td>\n",
    "<td>Очень медленный</td>\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Примеры"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для работы с русским языком в Yargy встроен морфологический анализатор <a href=\"http://pymorphy2.readthedocs.io/en/latest\">Pymorphy2</a>. Найдём в тексте топонимы, которые начинаются прилагательным и заканчиваются словами \"федерация\" или \"республика\". Например, \"Российская Федерация\", \"Донецкая народная республика\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Чеченской', 'республике']\n",
      "['Донецкая', 'народная', 'республика']\n"
     ]
    }
   ],
   "source": [
    "from yargy import Parser, rule, and_\n",
    "from yargy.predicates import gram, is_capitalized, dictionary\n",
    "\n",
    "\n",
    "GEO = rule(\n",
    "    and_(\n",
    "        gram('ADJF'),  # так помечается прилагательное, остальные пометки описаны в\n",
    "                       # http://pymorphy2.readthedocs.io/en/latest/user/grammemes.html\n",
    "        is_capitalized()\n",
    "    ),\n",
    "    gram('ADJF').optional().repeatable(),\n",
    "    dictionary({\n",
    "        'федерация',\n",
    "        'республика'\n",
    "    })\n",
    ")\n",
    "\n",
    "\n",
    "parser = Parser(GEO)\n",
    "text = '''\n",
    "В Чеченской республике на день рождения ...\n",
    "Донецкая народная республика провозгласила ...\n",
    "Башня Федерация — одна из самых высоких ...\n",
    "'''\n",
    "for match in parser.findall(text):\n",
    "    print([_.value for _ in match.tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Настоящие сложные грамматики для топонимов собраны в репозитории <a href=\"https://github.com/natasha/natasha\">Natasha</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Найти подстроку в тексте не достаточно, нужно разбить её на поля и нормализовать. Например, из фразы \"12 марта по приказу президента Владимира Путина ...\", извлечём объект `Person(position='президент', Name(first='Владимир', last='Путин'))`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Person(\n",
       "    position='президент',\n",
       "    name=Name(\n",
       "        first='владимир',\n",
       "        last='путин'\n",
       "    )\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from yargy import Parser\n",
    "from yargy.predicates import gram\n",
    "from yargy.pipelines import morph_pipeline\n",
    "from yargy.interpretation import fact\n",
    "from IPython.display import display\n",
    "\n",
    "Person = fact(\n",
    "    'Person',\n",
    "    ['position', 'name']\n",
    ")\n",
    "Name = fact(\n",
    "    'Name',\n",
    "    ['first', 'last']\n",
    ")\n",
    "\n",
    "\n",
    "POSITION = morph_pipeline([\n",
    "    'премьер министр',\n",
    "    'президент'\n",
    "])\n",
    "\n",
    "NAME = rule(\n",
    "    gram('Name').interpretation(\n",
    "        Name.first.inflected()\n",
    "    ),\n",
    "    gram('Surn').interpretation(\n",
    "        Name.last.inflected()\n",
    "    )\n",
    ").interpretation(\n",
    "    Name\n",
    ")\n",
    "\n",
    "PERSON = rule(\n",
    "    POSITION.interpretation(\n",
    "        Person.position.inflected()\n",
    "    ),\n",
    "    NAME.interpretation(\n",
    "        Person.name\n",
    "    )\n",
    ").interpretation(\n",
    "    Person\n",
    ")\n",
    "\n",
    "\n",
    "parser = Parser(PERSON)\n",
    "text = '''\n",
    "12 марта по приказу президента Владимира Путина ...\n",
    "'''\n",
    "for match in parser.findall(text):\n",
    "    display(match.fact)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Грамматики для имён собраны в репозитории [Natasha](https://github.com/natasha/natasha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Токенизатор"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Парсер работает с последовательностью токенов. Встроенный в Yargy токенизатор простой и предсказуемый:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Ростов', '-', 'на', '-', 'Дону']\n",
      "['Длительностью', '18', 'ч', '.', '10', 'мин', '.']\n",
      "['Яндекс', '.', 'Такси']\n",
      "['π', '≈', '3', '.', '1415']\n",
      "['1', '500', '000', '$']\n",
      "['http', ':', '/', '/', 'vk', '.', 'com']\n"
     ]
    }
   ],
   "source": [
    "from yargy.tokenizer import MorphTokenizer\n",
    "\n",
    "\n",
    "tokenizer = MorphTokenizer()\n",
    "text = '''Ростов-на-Дону\n",
    "Длительностью 18ч. 10мин.\n",
    "Яндекс.Такси\n",
    "π ≈ 3.1415\n",
    "1 500 000$\n",
    "http://vk.com\n",
    "'''\n",
    "for line in text.splitlines():\n",
    "    print([_.value for _ in tokenizer(line)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для каждого токена Pymorph2 возвращает набор граммем. Например, \"NOUN, sing, femn\" — \"существительное в единственном числе женского рода\". Полный список в <a href=\"https://pymorphy2.readthedocs.io/en/latest/user/grammemes.html\">документации Pymorph2</a>.\n",
    "\n",
    "Вне контекста слово имеет несколько вариантов разбора. Например, \"стали\" — глагол (VERB) во фразе \"мы стали лучше\" и существительное (NOUN) в \"марки стали\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[MorphToken(\n",
       "     value='марки',\n",
       "     span=[0, 5),\n",
       "     type='RU',\n",
       "     forms=[Form('марка', Grams(NOUN,femn,gent,inan,sing)),\n",
       "      Form('марка', Grams(NOUN,femn,inan,nomn,plur)),\n",
       "      Form('марка', Grams(NOUN,accs,femn,inan,plur)),\n",
       "      Form('маркий', Grams(ADJS,Qual,plur)),\n",
       "      Form('марк', Grams(NOUN,Name,anim,masc,nomn,plur))]\n",
       " ),\n",
       " MorphToken(\n",
       "     value='стали',\n",
       "     span=[6, 11),\n",
       "     type='RU',\n",
       "     forms=[Form('стать', Grams(VERB,indc,intr,past,perf,plur)),\n",
       "      Form('сталь', Grams(NOUN,femn,gent,inan,sing)),\n",
       "      Form('сталь', Grams(NOUN,femn,inan,nomn,plur)),\n",
       "      Form('сталь', Grams(NOUN,datv,femn,inan,sing)),\n",
       "      Form('сталь', Grams(NOUN,femn,inan,loct,sing)),\n",
       "      Form('сталь', Grams(NOUN,accs,femn,inan,plur))]\n",
       " )]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = MorphTokenizer()\n",
    "list(tokenizer('марки стали'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Токенизатор работает на правилах. В <a href=\"ref.ipynb#Токенизатор\">справочнике</a> показано, как менять стандартные правила и добавлять новые."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Предикаты"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предикат принимает токен, возвращает `True` или `False`. В Yargy встроен <a href=\"ref.ipynb#Предикаты\">набор готовых предикатов</a>. Операторы `and_`, `or_` и `not_` комбинируют предикаты:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yargy import and_, not_\n",
    "from yargy.tokenizer import MorphTokenizer\n",
    "from yargy.predicates import is_capitalized, eq\n",
    "\n",
    "\n",
    "tokenizer = MorphTokenizer()\n",
    "token = next(tokenizer('Стали'))\n",
    "\n",
    "predicate = is_capitalized()\n",
    "assert predicate(token) == True\n",
    "\n",
    "predicate = and_(\n",
    "    is_capitalized(),\n",
    "    not_(eq('марки'))\n",
    ")\n",
    "assert predicate(token) == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"ref.ipynb#predicates.custom\">custom</a> создаёт предикат из произвольной функции. Например, предикат для  римских цифр:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymorphy2.shapes import is_roman_number\n",
    "\n",
    "from yargy.parser import Context\n",
    "from yargy.tokenizer import Tokenizer\n",
    "from yargy.predicates import custom\n",
    "\n",
    "\n",
    "tokenizer = Tokenizer()\n",
    "token = next(tokenizer('XL'))\n",
    "\n",
    "predicate = custom(is_roman_number, types='LATIN')\n",
    "predicate = predicate.activate(Context(tokenizer))  # проверяется, что tokenizer поддерживает тип 'LATIN'\n",
    "assert predicate(token) == True\n",
    "\n",
    "token = next(tokenizer('XS'))\n",
    "assert predicate(token) == False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Газеттир"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Газеттир работает с последовательностью слов. Например, вместо:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yargy import or_, rule\n",
    "from yargy.predicates import normalized\n",
    "\n",
    "RULE = or_(\n",
    "    rule(normalized('dvd'), '-', normalized('диск')),\n",
    "    rule(normalized('видео'), normalized('файл'))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "удобно использовать `morph_pipeline`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Видео', 'файл']\n",
      "['dvd', '-', 'диске']\n"
     ]
    }
   ],
   "source": [
    "from yargy import Parser\n",
    "from yargy.pipelines import morph_pipeline\n",
    "\n",
    "\n",
    "RULE = morph_pipeline([\n",
    "    'dvd-диск',\n",
    "    'видео файл',\n",
    "    'видеофильм',\n",
    "    'газета',\n",
    "    'электронный дневник',\n",
    "    'эссе',\n",
    "])\n",
    "\n",
    "parser = Parser(RULE)\n",
    "text = 'Видео файл на dvd-диске'\n",
    "for match in parser.findall(text):\n",
    "    print([_.value for _ in match.tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Список газеттиров в <a href=\"ref.ipynb#Газеттир\">справочнике</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Грамматики"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В Yargy контекстно-свободная грамматика описывается конструкциями Python. Например, традиционная запись грамматики размеров одежды:\n",
    "```\n",
    "KEY -> р. | размер\n",
    "VALUE -> S | M | L\n",
    "SIZE -> KEY VALUE\n",
    "```\n",
    "\n",
    "Так она выглядит в Yargy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SIZE -> KEY VALUE\n",
       "KEY -> 'р' '.' | 'размер'\n",
       "VALUE -> 'S' | 'M' | 'L'\n"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from yargy import rule, or_\n",
    "\n",
    "\n",
    "KEY = or_(\n",
    "    rule('р', '.'),\n",
    "    rule('размер')\n",
    ").named('KEY')\n",
    "VALUE = or_(\n",
    "    rule('S'),\n",
    "    rule('M'),\n",
    "    rule('L'),\n",
    ").named('VALUE')\n",
    "SIZE = rule(\n",
    "    KEY,\n",
    "    VALUE\n",
    ").named('SIZE')\n",
    "SIZE.normalized.as_bnf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В Yargy терминал грамматики — предикат. Используем встроенный предикат `in_`, сократим запись `VALUE`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SIZE -> KEY VALUE\n",
       "KEY -> 'р' '.' | 'размер'\n",
       "VALUE -> in_(...)\n"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from yargy.predicates import in_\n",
    "\n",
    "\n",
    "VALUE = rule(\n",
    "    in_('SML')\n",
    ").named('VALUE')\n",
    "SIZE = rule(\n",
    "    KEY,\n",
    "    VALUE\n",
    ").named('SIZE')\n",
    "SIZE.normalized.as_bnf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Как быть, когда правая часть правила ссылается на левую? Например:\n",
    "```\n",
    "EXPR -> a | ( EXPR + EXPR )\n",
    "```\n",
    "\n",
    "В Python нельзя использовать необъявленные переменные. Для рекурсивных правил, есть конструкция `forward`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EXPR -> 'a' | '(' EXPR '+' EXPR ')'\n"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from yargy import forward\n",
    "\n",
    "\n",
    "EXPR = forward()\n",
    "EXPR.define(or_(\n",
    "    rule('a'),\n",
    "    rule('(', EXPR, '+', EXPR, ')')\n",
    ").named('EXPR'))\n",
    "EXPR.normalized.as_bnf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Рекурсивные правила описывают последовательности токенов произвольной длины. Грамматика для текста в кавычках:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TITLE -> '«' R0 '»'\n",
       "R0 -> not_('»') | not_('»') R0\n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from yargy import not_\n",
    "from yargy.predicates import eq\n",
    "\n",
    "\n",
    "WORD = not_(eq('»'))\n",
    "TEXT = forward()\n",
    "TEXT.define(or_(\n",
    "    rule(WORD),\n",
    "    rule(WORD, TEXT)\n",
    "))\n",
    "TITLE = rule(\n",
    "    '«',\n",
    "    TEXT,\n",
    "    '»'\n",
    ").named('TITLE')\n",
    "TITLE.normalized.as_bnf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для удобства в Yargy есть метод `repeatable` с ним запись короче. Библиотека автоматически добавит `forward`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TITLE -> '«' R0 '»'\n",
       "R0 -> not_('»') R0 | not_('»')\n"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TITLE = rule(\n",
    "    '«',\n",
    "    not_(eq('»')).repeatable(),\n",
    "    '»'\n",
    ").named('TITLE')\n",
    "TITLE.normalized.as_bnf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Парсер"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "У парсера есть два метода: `findall` и `match`. `findall` находит все непересекающиеся подстроки, которые удовлетворяют грамматике:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Президент', 'Владимир', 'Путин']\n",
      "['«', 'Интервью', 'с', 'Путиным', '»']\n"
     ]
    }
   ],
   "source": [
    "parser = Parser(\n",
    "    or_(\n",
    "        PERSON,\n",
    "        TITLE\n",
    "    )\n",
    ")\n",
    "text = 'Президент Владимир Путин в фильме «Интервью с Путиным» ..'\n",
    "for match in parser.findall(text):\n",
    "    print([_.value for _ in match.tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`match` — пытается разобрать весь текст целиком:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Президент', 'Владимир', 'Путин']\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "match = parser.match('Президент Владимир Путин')\n",
    "print([_.value for _ in match.tokens])\n",
    "\n",
    "match = parser.match('Президент Владимир Путин 25 мая')\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Интерпретация"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Результат работы парсера — это дерево разбора. Грамматика и деревья разбора для дат:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.2 (20221011.1828)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"88pt\" height=\"54pt\"\n",
       " viewBox=\"0.00 0.00 88.00 54.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 50)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-50 84,-50 84,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#aec7e8\" stroke=\"none\" points=\"70.5,-46 27.5,-46 27.5,-27 70.5,-27 70.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">DATE</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"38,-19 0,-19 0,0 38,0 38,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"19\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">2015</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M38.51,-26.76C36.37,-24.98 34.1,-23.08 31.89,-21.24\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"32.31,-20.23 29.33,-19.11 30.97,-21.84 32.31,-20.23\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"59,-19 39,-19 39,0 59,0 59,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">г</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M49,-26.76C49,-25.36 49,-23.9 49,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"50.05,-22.11 49,-19.11 47.95,-22.11 50.05,-22.11\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"80,-19 60,-19 60,0 80,0 80,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">.</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M56.34,-26.76C57.78,-25.05 59.29,-23.25 60.78,-21.48\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"61.64,-22.08 62.77,-19.11 60.03,-20.73 61.64,-22.08\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "Graph(nodes=[...], edges=[...])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.2 (20221011.1828)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"117pt\" height=\"54pt\"\n",
       " viewBox=\"0.00 0.00 116.50 54.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 50)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-50 112.5,-50 112.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#aec7e8\" stroke=\"none\" points=\"70,-46 27,-46 27,-27 70,-27 70,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"48.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">DATE</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"27,-19 0,-19 0,0 27,0 27,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"13.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">18</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M36.26,-26.76C33.66,-24.9 30.88,-22.92 28.2,-21\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"28.61,-20 25.56,-19.11 27.39,-21.71 28.61,-20\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"69,-19 28,-19 28,0 69,0 69,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"48.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">июля</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M48.5,-26.76C48.5,-25.36 48.5,-23.9 48.5,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"49.55,-22.11 48.5,-19.11 47.45,-22.11 49.55,-22.11\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"108.5,-19 70.5,-19 70.5,0 108.5,0 108.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"89.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">2016</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M62.84,-26.76C66.01,-24.82 69.41,-22.75 72.67,-20.76\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"73.36,-21.57 75.38,-19.11 72.27,-19.78 73.36,-21.57\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "Graph(nodes=[...], edges=[...])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.2 (20221011.1828)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"146pt\" height=\"54pt\"\n",
       " viewBox=\"0.00 0.00 145.50 54.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 50)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-50 141.5,-50 141.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#aec7e8\" stroke=\"none\" points=\"95.5,-46 52.5,-46 52.5,-27 95.5,-27 95.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"74\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">DATE</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"38,-19 0,-19 0,0 38,0 38,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"19\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">2016</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M55.07,-26.9C50.44,-24.79 45.45,-22.52 40.7,-20.36\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"40.94,-19.32 37.78,-19.03 40.07,-21.23 40.94,-19.32\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"59,-19 39,-19 39,0 59,0 59,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">&#45;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M65.26,-26.76C63.55,-25.05 61.75,-23.25 59.98,-21.48\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"60.48,-20.49 57.61,-19.11 58.99,-21.98 60.48,-20.49\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"87.5,-19 60.5,-19 60.5,0 87.5,0 87.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"74\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">01</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M74,-26.76C74,-25.36 74,-23.9 74,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"75.05,-22.11 74,-19.11 72.95,-22.11 75.05,-22.11\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"109,-19 89,-19 89,0 109,0 109,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">&#45;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M82.74,-26.76C84.45,-25.05 86.25,-23.25 88.02,-21.48\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"89.01,-21.98 90.39,-19.11 87.52,-20.49 89.01,-21.98\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"137.5,-19 110.5,-19 110.5,0 137.5,0 137.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"124\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">02</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>0&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M91.21,-26.9C96.45,-24.27 102.2,-21.4 107.4,-18.8\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"108.02,-19.66 110.23,-17.38 107.08,-17.79 108.02,-19.66\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "Graph(nodes=[...], edges=[...])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "from yargy.predicates import (\n",
    "    lte,\n",
    "    gte,\n",
    "    dictionary\n",
    ")\n",
    "\n",
    "\n",
    "MONTHS = {\n",
    "    'январь',\n",
    "    'февраль',\n",
    "    'март',\n",
    "    'апрель',\n",
    "    'мая',\n",
    "    'июнь',\n",
    "    'июль',\n",
    "    'август',\n",
    "    'сентябрь',\n",
    "    'октябрь',\n",
    "    'ноябрь',\n",
    "    'декабрь'\n",
    "}\n",
    "\n",
    "\n",
    "MONTH_NAME = dictionary(MONTHS)\n",
    "MONTH = and_(\n",
    "    gte(1),\n",
    "    lte(12)\n",
    ")\n",
    "DAY = and_(\n",
    "    gte(1),\n",
    "    lte(31)\n",
    ")\n",
    "YEAR = and_(\n",
    "    gte(1900),\n",
    "    lte(2100)\n",
    ")\n",
    "DATE = or_(\n",
    "    rule(DAY, MONTH_NAME, YEAR),\n",
    "    rule(YEAR, '-', MONTH, '-', DAY),\n",
    "    rule(YEAR, 'г', '.')\n",
    ").named('DATE')\n",
    "parser = Parser(DATE)\n",
    "text = '''2015г.\n",
    "18 июля 2016\n",
    "2016-01-02\n",
    "'''\n",
    "for line in text.splitlines():\n",
    "    match = parser.match(line)\n",
    "    display(match.tree.as_dot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Интерпретация — процесс преобразования дерева разбора в объект с набором полей. Для даты, например, нужно получить структуры вида `Date(year=2016, month=1, day=2)`. Пользователь размечает дерево на вершины-атрибуты и вершины-конструкторы методом `interpretation`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.2 (20221011.1828)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"110pt\" height=\"81pt\"\n",
       " viewBox=\"0.00 0.00 109.50 81.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 77)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-77 105.5,-77 105.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"89.5,-73 51.5,-73 51.5,-54 89.5,-54 89.5,-73\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.5\" y=\"-61\" font-family=\"sans\" font-size=\"10.00\">Date</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"59,-46 0,-46 0,-27 59,-27 59,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"29.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.year</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M56.16,-53.76C52.99,-51.82 49.59,-49.75 46.33,-47.76\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"46.73,-46.78 43.62,-46.11 45.64,-48.57 46.73,-46.78\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"80.5,-46 60.5,-46 60.5,-27 80.5,-27 80.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">г</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M70.5,-53.76C70.5,-52.36 70.5,-50.9 70.5,-49.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"71.55,-49.11 70.5,-46.11 69.45,-49.11 71.55,-49.11\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"101.5,-46 81.5,-46 81.5,-27 101.5,-27 101.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"91.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">.</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M77.84,-53.76C79.28,-52.05 80.79,-50.25 82.28,-48.48\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"83.14,-49.08 84.27,-46.11 81.53,-47.73 83.14,-49.08\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"48.5,-19 10.5,-19 10.5,0 48.5,0 48.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"29.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">2015</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M29.5,-26.76C29.5,-25.36 29.5,-23.9 29.5,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"30.55,-22.11 29.5,-19.11 28.45,-22.11 30.55,-22.11\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "Graph(nodes=[...], edges=[...])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.2 (20221011.1828)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"194pt\" height=\"81pt\"\n",
       " viewBox=\"0.00 0.00 193.50 81.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 77)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-77 189.5,-77 189.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"110,-73 72,-73 72,-54 110,-54 110,-73\"/>\n",
       "<text text-anchor=\"middle\" x=\"91\" y=\"-61\" font-family=\"sans\" font-size=\"10.00\">Date</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"56,-46 0,-46 0,-27 56,-27 56,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"28\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.day</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M71.76,-54.86C65.78,-52.49 59.12,-49.85 52.82,-47.35\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"52.95,-46.27 49.77,-46.14 52.17,-48.22 52.95,-46.27\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"125,-46 57,-46 57,-27 125,-27 125,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"91\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.month</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M91,-53.76C91,-52.36 91,-50.9 91,-49.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"92.05,-49.11 91,-46.11 89.95,-49.11 92.05,-49.11\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"185.5,-46 126.5,-46 126.5,-27 185.5,-27 185.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"156\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.year</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;5 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>0&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M110.15,-55.13C116.57,-52.66 123.84,-49.87 130.66,-47.25\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"131.13,-48.19 133.55,-46.13 130.37,-46.23 131.13,-48.19\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"41.5,-19 14.5,-19 14.5,0 41.5,0 41.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"28\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">18</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M28,-26.76C28,-25.36 28,-23.9 28,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"29.05,-22.11 28,-19.11 26.95,-22.11 29.05,-22.11\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"111.5,-19 70.5,-19 70.5,0 111.5,0 111.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"91\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">июля</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;4 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>3&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M91,-26.76C91,-25.36 91,-23.9 91,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"92.05,-22.11 91,-19.11 89.95,-22.11 92.05,-22.11\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"175,-19 137,-19 137,0 175,0 175,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"156\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">2016</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M156,-26.76C156,-25.36 156,-23.9 156,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"157.05,-22.11 156,-19.11 154.95,-22.11 157.05,-22.11\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "Graph(nodes=[...], edges=[...])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.2 (20221011.1828)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"236pt\" height=\"81pt\"\n",
       " viewBox=\"0.00 0.00 235.50 81.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 77)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-77 231.5,-77 231.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"134.5,-73 96.5,-73 96.5,-54 134.5,-54 134.5,-73\"/>\n",
       "<text text-anchor=\"middle\" x=\"115.5\" y=\"-61\" font-family=\"sans\" font-size=\"10.00\">Date</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"59,-46 0,-46 0,-27 59,-27 59,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"29.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.year</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M96.41,-56.95C86.29,-54.01 73.56,-50.31 61.94,-46.93\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"62.18,-45.91 59.01,-46.08 61.6,-47.92 62.18,-45.91\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"80.5,-46 60.5,-46 60.5,-27 80.5,-27 80.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">&#45;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;3 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M99,-53.94C94.28,-51.43 89.16,-48.65 84.5,-46 84.18,-45.82 83.85,-45.63 83.53,-45.44\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"83.67,-44.31 80.55,-43.7 82.61,-46.12 83.67,-44.31\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"149.5,-46 81.5,-46 81.5,-27 149.5,-27 149.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"115.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.month</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M115.5,-53.76C115.5,-52.36 115.5,-50.9 115.5,-49.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"116.55,-49.11 115.5,-46.11 114.45,-49.11 116.55,-49.11\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"170.5,-46 150.5,-46 150.5,-27 170.5,-27 170.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"160.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">&#45;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;6 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M132,-53.94C136.72,-51.43 141.84,-48.65 146.5,-46 146.82,-45.82 147.15,-45.63 147.47,-45.44\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"148.39,-46.12 150.45,-43.7 147.33,-44.31 148.39,-46.12\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<polygon fill=\"#dbdb8d\" stroke=\"none\" points=\"227.5,-46 171.5,-46 171.5,-27 227.5,-27 227.5,-46\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.5\" y=\"-34\" font-family=\"sans\" font-size=\"10.00\">Date.day</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;7 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>0&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M134.56,-56.83C144.47,-53.88 156.84,-50.2 168.12,-46.84\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"168.69,-47.77 171.27,-45.9 168.09,-45.75 168.69,-47.77\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"48.5,-19 10.5,-19 10.5,0 48.5,0 48.5,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"29.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">2016</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M29.5,-26.76C29.5,-25.36 29.5,-23.9 29.5,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"30.55,-22.11 29.5,-19.11 28.45,-22.11 30.55,-22.11\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"129,-19 102,-19 102,0 129,0 129,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"115.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">01</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M115.5,-26.76C115.5,-25.36 115.5,-23.9 115.5,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"116.55,-22.11 115.5,-19.11 114.45,-22.11 116.55,-22.11\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"213,-19 186,-19 186,0 213,0 213,-19\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.5\" y=\"-7\" font-family=\"sans\" font-size=\"10.00\">02</text>\n",
       "</g>\n",
       "<!-- 7&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>7&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"gray\" d=\"M199.5,-26.76C199.5,-25.36 199.5,-23.9 199.5,-22.45\"/>\n",
       "<polygon fill=\"gray\" stroke=\"gray\" points=\"200.55,-22.11 199.5,-19.11 198.45,-22.11 200.55,-22.11\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "Graph(nodes=[...], edges=[...])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from yargy.interpretation import fact\n",
    "\n",
    "\n",
    "Date = fact(\n",
    "    'Date',\n",
    "    ['year', 'month', 'day']\n",
    ")\n",
    "\n",
    "\n",
    "DATE = or_(\n",
    "    rule(\n",
    "        DAY.interpretation(\n",
    "            Date.day\n",
    "        ),\n",
    "        MONTH_NAME.interpretation(\n",
    "            Date.month\n",
    "        ),\n",
    "        YEAR.interpretation(\n",
    "            Date.year\n",
    "        )\n",
    "    ),\n",
    "    rule(\n",
    "        YEAR.interpretation(\n",
    "            Date.year\n",
    "        ),\n",
    "        '-',\n",
    "        MONTH.interpretation(\n",
    "            Date.month\n",
    "        ),\n",
    "        '-',\n",
    "        DAY.interpretation(\n",
    "            Date.day\n",
    "        )\n",
    "    ),\n",
    "    rule(\n",
    "        YEAR.interpretation(\n",
    "            Date.year\n",
    "        ),\n",
    "        'г', '.'\n",
    "    )\n",
    ").interpretation(\n",
    "    Date\n",
    ").named('DATE')\n",
    "\n",
    "parser = Parser(DATE)\n",
    "for line in text.splitlines():\n",
    "    match = parser.match(line)\n",
    "    display(match.tree.as_dot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Из размеченного дерева библиотека собирает объект:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date(\n",
       "    year='2015',\n",
       "    month=None,\n",
       "    day=None\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Date(\n",
       "    year='2016',\n",
       "    month='июля',\n",
       "    day='18'\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Date(\n",
       "    year='2016',\n",
       "    month='01',\n",
       "    day='02'\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for line in text.splitlines():\n",
    "    match = parser.match(line)\n",
    "    display(match.fact)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подробнее об интерпретации в <a href=\"#ref.ipynb#Интерпретация\">справочнике</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Нормализация"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Содержание полей фактов нужно нормировать. Например, не `Date('июня', '2018')`, а `Date(6, 2018)`; не `Person('президента', Name('Владимира', 'Путина'))`, а `Person('президент', Name('Владимир', 'Путин'))`. В Yargy пользователь при разметке дерева разбора указывает, как нормировать вершины-атрибуты. В примере слово \"июня\" будет приведено к нормальной форме \"июнь\" и заменится на число \"6\" с помощью словаря `MONTHS`. Год и день просто приводятся к `int`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date(\n",
       "    year=2016,\n",
       "    month=6,\n",
       "    day=18\n",
       ")"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MONTHS = {\n",
    "    'январь': 1,\n",
    "    'февраль': 2,\n",
    "    'март': 3,\n",
    "    'апрель': 4,\n",
    "    'мая': 5,\n",
    "    'июнь': 6,\n",
    "    'июль': 7,\n",
    "    'август': 8,\n",
    "    'сентябрь': 9,\n",
    "    'октябрь': 10,\n",
    "    'ноябрь': 11,\n",
    "    'декабрь': 12\n",
    "}\n",
    "\n",
    "\n",
    "DATE = rule(\n",
    "    DAY.interpretation(\n",
    "        Date.day.custom(int)\n",
    "    ),\n",
    "    MONTH_NAME.interpretation(\n",
    "        Date.month.normalized().custom(MONTHS.get)\n",
    "    ),\n",
    "    YEAR.interpretation(\n",
    "        Date.year.custom(int)\n",
    "    )\n",
    ").interpretation(\n",
    "    Date\n",
    ")\n",
    "\n",
    "parser = Parser(DATE)\n",
    "match = parser.match('18 июня 2016')\n",
    "match.fact"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подробнее в <a href=\"ref.ipynb#Нормализация\">справочнике</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Согласование"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Примитивная грамматика имён:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = rule(\n",
    "    gram('Name').interpretation(\n",
    "        Name.first.inflected()\n",
    "    ),\n",
    "    gram('Surn').interpretation(\n",
    "        Name.last.inflected()\n",
    "    )\n",
    ").interpretation(\n",
    "    Name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "У неё есть две проблемы. Она срабатывает на словосочетаниях, где имя и фамилия в разных падежах:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Илье', 'Ивановым']\n",
      "['Павлом', 'Семенов']\n"
     ]
    }
   ],
   "source": [
    "parser = Parser(NAME)\n",
    "for match in parser.findall('Илье Ивановым, Павлом Семенов'):\n",
    "    print([_.value for _ in match.tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Имя и фамилия приводятся к нормальной форме независимо, получается женщина \"Иванов\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name(\n",
       "    first='саша',\n",
       "    last='иванов'\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Name(\n",
       "    first='саша',\n",
       "    last='иванов'\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "parser = Parser(NAME)\n",
    "for match in parser.findall('Сашу Иванову, Саше Иванову'):\n",
    "    display(match.fact)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В Yargy связь между словами и словосочетаниями устанавливается методом `match`. Для согласования по числу в `match` передаём `number_relation`, для согласования по падежу, роду и числу — `gnc_relation`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yargy.relations import gnc_relation\n",
    "\n",
    "\n",
    "gnc = gnc_relation()\n",
    "\n",
    "NAME = rule(\n",
    "    gram('Name').interpretation(\n",
    "        Name.first.inflected()\n",
    "    ).match(gnc),\n",
    "    gram('Surn').interpretation(\n",
    "        Name.last.inflected()\n",
    "    ).match(gnc)\n",
    ").interpretation(\n",
    "    Name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Саша', 'Быков']\n"
     ]
    }
   ],
   "source": [
    "parser = Parser(NAME)\n",
    "for match in parser.findall('Илье Ивановым, Павлом Семенов, Саша Быков'):\n",
    "    print([_.value for _ in match.tokens])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name(\n",
       "    first='саша',\n",
       "    last='иванова'\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Name(\n",
       "    first='саша',\n",
       "    last='иванов'\n",
       ")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "parser = Parser(NAME)\n",
    "for match in parser.findall('Сашу Иванову, Саше Иванову'):\n",
    "    display(match.fact)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Список согласований в <a href=\"ref.ipynb#Согласование\">справочнике</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
